The work is branched from the other not yet merged integration test PR #191
The API based test is just the last commit.

Will update once #191 is merged.

@rajesh-gali here I tried to write up the problem I was facing during the development of the test suite (CLI and CAPI) with usage of the config file

Provider double-init / deadlock: the core problem and how the test suites avoid it

What activate = 1 does in openssl.cnf

The config has activate = 1 in the [azihsm_sect] section. This tells libcrypto to load and initialize the provider immediately during config processing — the application never calls OSSL_PROVIDER_load() itself. The internal call chain inside libcrypto:

OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG, ...)
  → ossl_init_config_settings()                        // RUN_ONCE macro
    → ossl_config_int()
      → CONF_modules_load_file()                       // reads OPENSSL_CONF env var
        → provider_conf_load()                         // checks activate=1
          → ossl_provider_activate()
            → provider_init()
              → DSO_load() + DSO_bind_func("OSSL_provider_init")
                → provider's OSSL_provider_init()      // entry point in azihsm_provider.so

Config loading is triggered either by an explicit OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG, ...) call, or implicitly when provider operations (e.g. ossl_provider_find()) trigger config loading internally. The openssl CLI binary calls it explicitly during startup.

Without activate = 1, the section just declares the provider — it's only loaded when explicitly requested. With it, provider_conf_load() checks the activate key and calls ossl_provider_activate() to load the provider immediately. This is the standard way to make a provider available to CLI tools like openssl genpkey, which don't have their own code to call OSSL_PROVIDER_load().

The core problem

When OpenSSL's default library context auto-loads this config (triggered by the OPENSSL_CONF env var), it loads and initializes the azihsm provider into the default context.

The deadlock occurs when the provider's code uses OpenSSL API functions internally (EVP_MD_fetch, crypto primitives, etc.). If the provider embeds its own copy of libcrypto (static linking), that second libcrypto instance auto-initializes on first use — sees OPENSSL_CONF in the environment, reads the config, encounters activate = 1, and attempts to load the provider again. Recursive init, deadlock.

The provider doesn't explicitly call OpenSSL initialization — it's the auto-init mechanism in the statically-linked libcrypto copy that triggers it. Any OpenSSL API call from the provider is enough to start the chain.

How each suite avoids it

CLI tests — shared OpenSSL (single libcrypto)

With shared linking, the openssl binary and azihsm_provider.so share the same libcrypto.so.3 via the dynamic linker. The dynamic linker loads it once. When the binary's libcrypto reads openssl.cnf and loads the provider, the provider's calls into libcrypto go to the already-initialized single instance — no second auto-init, no recursion.

This is why the CI deadlocked before: OpenSSL was built with no-shared -fvisibility=hidden, creating two separate static copies of libcrypto. The fix was building OpenSSL with shared libraries (.github/workflows/rust.yml).

CAPI tests — OPENSSL_INIT_NO_LOAD_CONFIG + explicit context

The CAPI tests use a two-layer defense:

1. OPENSSL_init_crypto(OPENSSL_INIT_NO_LOAD_CONFIG, NULL) (plugins/ossl_prov/integration-tests/openssl-capi/cpp/main.cpp:13) — suppresses automatic config loading on the default context. This skips the config loading step entirely (runs ossl_init_no_config() instead, which marks config as "done" without loading any file), so even though OPENSSL_CONF is set, the default context never reads it and the provider is never loaded into the default context.

2. OSSL_LIB_CTX_new() + OSSL_LIB_CTX_load_config() (plugins/ossl_prov/integration-tests/openssl-capi/cpp/utils/provider_ctx.hpp:25-44) — each test creates a dedicated, non-default library context and explicitly loads the config into it. The provider loads into this isolated context only. Since the default context never loaded the provider, there's no double-init.

Summary

The CAPI approach is inherently safe regardless of static/shared linking because it never lets the default context auto-load the provider. The CLI approach depends on shared linking to avoid the recursive init.

Annotated call stacks

CLI test suite

1. Rust test runner (nextest) invokes bash script
   e.g. plugins/ossl_prov/integration-tests/openssl-cli/testfiles/ec/round_trip/round_trip.sh

2. Script sources env.sh
   plugins/ossl_prov/integration-tests/openssl-cli/testfiles/env.sh
   ├── Generates key material in $REPO_ROOT/target/test-keymat/cli/     (lines 59-87)
   ├── Generates openssl.cnf with activate=1 for azihsm provider        (lines 95-119)
   │   └── module = <absolute path to azihsm_provider.so>
   │   └── activate = 1
   └── export OPENSSL_CONF="$AZIHSM_KEY_DIR/openssl.cnf"                (line 121)

3. Script calls $OPENSSL_BIN (e.g. genpkey, dgst)
   └── openssl binary starts
       └── libcrypto.so.3 config loading (OPENSSL_INIT_LOAD_CONFIG)
           └── reads OPENSSL_CONF
               └── activate=1 → OSSL_PROVIDER_load("azihsm")
                   └── dlopen(azihsm_provider.so)
                       └── OSSL_provider_init()
                           └── provider uses EVP_* calls → same libcrypto.so.3
                               (shared linking: single instance, already initialized,
                                no recursion)

4. openssl CLI executes the cryptographic operation (genpkey/dgst/req)
   using the now-loaded azihsm provider via -propquery "?provider=azihsm"

Key requirement: OpenSSL must be built with shared libraries so the binary and provider share one libcrypto.so.3. CI builds OpenSSL with ./Configure --prefix=/opt/openssl-3.0.3 --libdir=lib -fPIC (.github/workflows/rust.yml:122-123).

CAPI test suite

1. Rust test runner discovers gtest binary and enumerates tests
   plugins/ossl_prov/integration-tests/openssl-capi/cpp/openssl_capi_integration_tests.rs
   ├── Generates key material in $REPO_ROOT/target/test-keymat/capi/     (generate_dev_key_material())
   ├── Generates openssl.cnf with activate=1                             (generate_openssl_conf())
   └── Calls gtest binary with --gtest_list_tests, parses output         (parse_gtest_list())
       └── Passes OPENSSL_CONF to each subprocess via .env()             (lines 336, 408)

2. For each test, Rust runner invokes the gtest binary with --gtest_filter=Suite.Test

3. gtest binary main() runs BEFORE any test
   plugins/ossl_prov/integration-tests/openssl-capi/cpp/main.cpp:13
   └── OPENSSL_init_crypto(OPENSSL_INIT_NO_LOAD_CONFIG, NULL)
       └── Marks default context: SKIP config auto-loading
           (OPENSSL_CONF is in env but default context will never read it)

4. Individual test creates ProviderCtx (RAII)
   plugins/ossl_prov/integration-tests/openssl-capi/cpp/utils/provider_ctx.hpp:23-68
   ├── OSSL_LIB_CTX_new()                                                (line 25)
   │   └── Creates a fresh, isolated library context (not the default)
   ├── std::getenv("OPENSSL_CONF")                                       (line 35)
   └── OSSL_LIB_CTX_load_config(libctx_, conf)                           (line 44)
       └── Loads config INTO the isolated context only
           └── activate=1 → OSSL_PROVIDER_load("azihsm") into libctx_
               └── Provider initializes in isolated context
                   └── Provider's OpenSSL API calls use the process-wide libcrypto
                       but the DEFAULT context was never loaded with provider config
                       → no recursive activation, no deadlock

5. Test runs cryptographic operations against ProviderCtx::libctx()
   e.g. EVP_PKEY_CTX_new_from_name(libctx, "EC", "?provider=azihsm")

6. ProviderCtx destructor (~ProviderCtx)                                 (line 70-76)
   └── OSSL_LIB_CTX_free(libctx_)
       └── Unloads provider, releases all resources for that context

Key requirement: OPENSSL_INIT_NO_LOAD_CONFIG must be called before any other OpenSSL API use. This is why it's in main() before InitGoogleTest. The per-test OSSL_LIB_CTX isolation also ensures tests don't interfere with each other.

Closed with respect to #266